​

​

Problem statement:

Increase free to paid conversion by 10%

​

Overview:

The current free to paid conversation rate of Bumble is 9%. The current features exposed to only a paid user are sending unlimited likes, see who likes you, send a compliment etc.

This experiment explores if a new feature( bumble cafe- an offline social mixer) improves the conversion rate further.

​

Hypothesis:

If we introduce a offline bumble mixer called bumble cafe to premium users, then our free to paid conversion will increase by 10% due to increase in certainty of meeting someone through bumble.

Null hypothesis: Showing premium only feature of bumble cafe will not improve retention rate of free to paid

​

​

Goal:

Revenue = number of active users * conversion from free to paid users * ARPU

The experiment is being run on input lever of increase conversion rate and thus it directly impact the business objective for Bumble

​

Success Metrics:

Worst case scenario: Absolute increase in conversion by 3%

Best case scenario: Absolute increase in conversion by 10%

health metrics: Average number of sessions and session depth per user

​

Experiment design:

1. What are we testing: We are testing if adding a new premium feature of "bumble cafe" will increase conversion rate of free to paid due to increase in certainty of meeting a match.
   
   Support evidence: User calling.
   
   Qualitative data that supports the hypothesis. 30 user interviews are conducted with bumble free users and below are the results
   
   Top reasons why free users aren't converting - No certainty of converting a match into real meet, Paradox of choices swiping too many profiles, No lazy pay or pay later options
   
   The experiment is based on firth insight
   ​
2. Variation design
   
   Control group flow:
   
   free user swipes daily limit --> asked to upgrade to premium --> redirected to profile page for premium plans --> drops off to remain free user
   
   
   
   Variation A expected flow:
   
   free user discovering new feature bumble cafe --> user clicks on the icon --> user is
   re-directed to details page --> CTA to upgrade
   
   
   
   
3. Audience & Sample size
   
   Segmentation:
   Male free users - who get less than 3 matches per week
   Female free users - who have been active on bumble for more than 3 weeks.
   
   Sample size:
   
   Control: 15000
   Variation A: 15000
   
   Duration of test:
   May 25th, 2024 to June 25th, 2024 or till we are statistically significant; whichever is earlier
   
   

​

​

4. A/A test implementation:
   
   
   

​

​

Experiment results:

From the data above, we can see that more users are discovering the upgrade page due to the new feature. Also, the conversion rates have been increased.

However, the health metric of average time spent by user per day has decreased. This is because more users are spending less time swiping and opting to be part of offline bumble mixer instead. This is concerning because this could affect bumble ad revenue negatively.

​

Release Decision:

​

Kill the experiment as it was unsuccessful with statistical significance. The experiment was unsuccessful as it negatively affected the health metrics.

Learnings:

• Experiments have to be designed taking into consideration all components of business objective. In this experiment, the ad-revenue impact was not considered
• Online dating could have seasonal affect. For example, people could be more willing to sign up for a offline mixer during summers than during winters. Therefore, experiments should be longer to cover the full cycle

​

Next steps:

Other null hypothesis we could experiment on

• Showing premium only feature of bumble cafe in the profile page instead of home page will not improve retention rate
• Make it a mandate for free users to unlock bumble cafe by spending more time on platform will not positively affect overall success and health metrics

Stakeholder management:

​